Hard to distinguish effects of specific policies from variables that cause those policies to be implemented.
But arbitrary cutoffs are extremely common in policy: “need” may be a complex latent characteristic that is nearly impossible to measure, but “is your household income less than 130% of the federal poverty line?” is just a check box.
We need a running variable and a discontinuity/cutoff. The running variable must be ordinal/continuous
The cutoff should impact our outcome only through its impact on the IV of interest
Units near the cutoff should be as similar as possible
The amount of random variation shouldn’t suddenly increase near the cutoff
Observational research suggests compliance is quite low.
But this is a federal law, so we don’t have a reasonable comparison group.
However, the age cutoff itself could be a source of a discontinuity.
The simplest way to estimate the effect is a linear model with a control for the running variable and an indicator for the treatment. (we can also center the “age” at 21 to simplify the interpretation
| model 1 | |
|---|---|
| constant | 91.841 |
| [90.220, 93.463] | |
| age | −0.975 |
| [−2.249, 0.299] | |
| over 21? | 7.663 |
| [4.762, 10.564] | |
| Num.Obs. | 48 |
| R2 | 0.595 |
| Age centered at 0 = 21 |
That said, this probably isn’t the best method because the slopes are different before and after the cutoff.
Using an interaction term allows the slope of the line to be different before and after the cutoff.
| model 1 | model 2 | |
|---|---|---|
| constant | 91.841 | 93.618 |
| [90.220, 93.463] | [91.739, 95.498] | |
| age | −0.975 | 0.827 |
| [−2.249, 0.299] | [−0.823, 2.477] | |
| over 21? | 7.663 | 7.663 |
| [4.762, 10.564] | [5.005, 10.320] | |
| age x over 21 | −3.603 | |
| [−5.937, −1.269] | ||
| Num.Obs. | 48 | 48 |
| R2 | 0.595 | 0.668 |
| Age centered at 0 = 21 |
However, the dis-aggregated results suggest that this isn’t the case: its more likely there are two separate trends.
One important consideration here is accounting for non-linearity. Properly controlling for the effect of the running variable requires us to get the functional form right.
Assuming a linear effect when the effect is cubic will produce a spurious result. Note that there appears to be a discontinuity here when using a linear model.
Adding polynomials (squared, cubed, etc versions of the running variable) is one option, but
can result in over-fitting
still can’t approximate certain kinds of non-linearity
Methods such as LOESS operate by estimating weighted polynomial regressions on a sliding “window” (usually called the bandwidth) of data points and then smoothing that result using a kernel function*.
All else equal, a smaller window gives greater weight to individual points, while a larger window will result in a smoother line.
*as you might guess, this is related to the methods that generate kernel density plots
Is the incumbency advantage real? How could we estimate it? Are there reasons to doubt it exists?
Obviously, there’s a built-in discontinuity here!
The relationship is a bit easier to spot if we focus on observations near the cutoff point, but still messy, and good reason to suspect the effect is non-linear
Using rdrobust to estimate an optimal bin size:
A simple test for the assumptions for RDD is to set a different cutoff and see if the results are still significant.
| Coeff | Std..Err. | z | ci.CI.Lower | ci.CI.Upper |
|---|---|---|---|---|
| 2.12 | 2.29 | 0.927 | -2.36 | 6.6 |
Vote share is the most common design for political scientists
Cattaneo, Frandsen, and Titiunik (2014): the incumbency advantage for U.S. Senators
Meyerson (2014): election of Islamic political parties and educational attainment for women
Pettersson-Lidbom (2008): election of left-wing governments in Sweden and unemployment/taxation
Technically, we’re estimating a local average treatment effect near the cutoff. This only matches the average treatment effect if the effects are similar across all values of the running variable. So consider:
People near an income threshold undoubtedly benefit more from income assistance compared to people far away
Swing districts may be more economically volatile than more partisan ones.
People around age 21 may do riskier stuff when they drink compared to older people.
Whether this matters is partly a question of interpretation.
Plurality elections impose a sharp boundary, but other boundaries may be “soft”
Not everyone who is eligible for food stamps will enroll
Some elected officials resign or never take their seats
The thresholds themselves may be measured with error (blood alcohol content, registration, place of residence)
This could be considered similar to the problem of “non-compliance”: some units should be treated, but aren’t. Methods are similar: estimate a 2 stage regression model to adjust for non-compliance
Hopkins (2004): county threshold for language minorities and ballot assistance
Posner: Regression discontinuity and cultural cleavage
Are observations near the cutoff fundamentally unlike observations elsewhere?
Is it possible some cases are changing their behavior in response to being “near” the cutoff?
Do small changes to the method for fitting the running variable cause big changes in the estimated effect? If so, the results might be spurious.
Does a placebo test give null results?
Testing “fake” cutoffs shortly before or after the real one
Checking for discontinuities on other variables that seem like they shouldn’t be responsive to the treatment (i.e. do homicides also spiking around 21 might indicate a different process)